Android Benchmark

Every time a new phone hits the market, and often even before that, it is run through countless numbers of synthetic benchmarks that showcase its computing prowess. People rarely talk about real-world performance, and if they do, it is either anecdotal (“The phone feels very snappy”, “Only occasional hick-ups”, “Apps launch immediately”), or in the form of manual side-by-side app-launching comparisons on YouTube. At the time of writing, there are, to the best of our knowledge, no real-world performance benchmark applications on Google Play. We present the first such application, called DiscoMark, which is now freely available on Google Play and does not require root access. DiscoMark uses the standard AccessibiltyService Android-API to measure the launch-times of applications; an actual real-world performance metric. Users can select which applications they want to include in the benchmark and how often the test should be run for averaging. DiscoMark then presents the average result for each app, and allows exporting more detailed results in CSV-format for further analysis. Furthermore, it allows users to compare their own results to those of our other users and gives tips on how to improve performance. DiscoMark can also be used to gain a better understanding of real-world performance and to figure out what factors influence it the most. For example, one can easily determine the effect that clearing all recent apps, or rebooting the phone has on performance. We show results of several such experiments that were performed using DiscoMark. We found, for example, that the battery saver mode on the Samsung Galaxy S6 does not save any battery during active use, but reduces performance by about 50%. We also analysed the influence that the installing of new apps has on phone-performance, and compared the performance of different versions of Android on a Nexus 5. After a successful release and promotion, many people downloaded DiscoMark and we managed to gather data from over 5000 users and more than 1000 different phones. We gained many insights into real-world performance from analysing the collected user-data. Among other things, we discovered that uninstalling the Facebook-App brings an average speed-up of 19% and that the OnePlus One bests all other phones in real-world performance, especially when apps are already in memory (hot-starts). Overall, we found that synthetic benchmarks are a useful indicator of real-world performance, i.e., on average, newer and faster phones tend to perform better in the real world. However, these benchmarks only reveal a small part of the entire picture, and the benefits of real-world benchmarking are manifold.